Whither Music IR Evaluation Infrastructure: Lessons to be Learned from TREC

Benchmark tasks are a powerful method for advancing the state of the art in a field. The music information retrieval community recently acknowledged the utility of such tasks by resolving to create an evaluation framework for music information retrieval (MIR) tasks and music digital libraries (MDL). This paper describes the processes used in the Text REtrieval Conference (TREC) evaluations to create information retrieval evaluation infrastructure, reviews assessments of how appropriate the evaluation methodology is for TREC tasks, and makes suggestions regarding the development of an MIR/MDL evaluation framework based on TREC experience.